Greptile Summary

This PR significantly improves the performance of FairSchedulingAlgoContext creation by replacing an unconditional txn.GetAll() call (which iterated every job in the database) with a targeted fetch of only the jobs relevant to the pool being scheduled: all currently leased jobs, all terminal jobs (for short-job penalty calculation), and queued jobs scoped to the home/away pools under consideration.

Key changes:

• New Leased() predicate on Job — !queued && !terminal && LatestRun() != nil — cleanly encapsulates the leased-job concept used throughout the new indexes.
• Two new secondary indexes in JobDb/Txn (leasedJobs, terminalJobs) maintained incrementally in Upsert and delete, following the same immutable-set pattern already used for unvalidatedJobs.
• GetAllLeasedJobs(), GetAllTerminalJobs(), GetQueuedJobsByPool(pool) — new query methods that expose the new indexes and the existing jobsByPoolAndQueue map.
• Generic UniqueBy utility in common/slices to deduplicate the assembled allJobs slice (necessary because a job queued for multiple pools in allPools would otherwise appear more than once).
• Previously flagged review issues (copy-paste bug assigning newLeasedJobs to txn.unvalidatedJobs, and over-allocation in GetQueuedJobsByPool) are both correctly addressed in this revision.
• The optimization is logically sound: queued jobs for pools outside allPools were already no-ops inside calculateJobSchedulingInfo (filtered by pool membership), so excluding them at load time produces identical scheduling results.
• Minor: terminal jobs are loaded globally rather than per-pool; the short-job penalty filter inside calculateJobSchedulingInfo already gates by jobPool == currentPool, so correctness is unaffected, but in systems with many completed jobs this remains an optimisation opportunity noted in the PR description.

Confidence Score: 5/5

Safe to merge — the optimization is logically equivalent to the previous full-scan approach, previously flagged bugs are fixed, and the new indexes are well-tested.

All remaining findings are P2: a missing capacity hint in UniqueBy and a slightly inaccurate code comment. Neither affects correctness or runtime behaviour. The previously raised P0/P1 concerns (copy-paste bug in delete, over-allocation in GetQueuedJobsByPool) are both resolved in this revision.

No files require special attention.

Important Files Changed

Sequence Diagram

sequenceDiagram
    participant SA as FairSchedulingAlgo
    participant TXN as JobDb Txn
    participant CSI as calculateJobSchedulingInfo

    SA->>TXN: GetAllLeasedJobs()
    TXN-->>SA: leased []Job

    SA->>TXN: GetAllTerminalJobs()
    TXN-->>SA: terminal []Job

    loop for each pool in allPools
        SA->>TXN: GetQueuedJobsByPool(pool)
        TXN-->>SA: queued []Job
    end

    SA->>SA: UniqueBy(allJobs, job.Id())

    SA->>CSI: calculateJobSchedulingInfo(allJobs, ...)
    CSI-->>SA: jobSchedulingInfo (demand, allocation, shortJobPenalty, jobsByPool)

    SA->>SA: build nodeDb from leasedJobs (jobsByPool) using currentPoolJobs + otherPoolsJobs
    SA-->>SA: FairSchedulingAlgoContext

_{Reviews (8): Last reviewed commit: "Merge branch 'master' into improve_fsctx..." | Re-trigger Greptile}